Optimized photorealistic audiovisual speech synthesis using active appearance modeling

Active appearance models can represent image information in terms of shape and texture parameters. This paper explains why this makes them highly suitable for data-based 2D audiovisual text-to-speech synthesis. We elaborate on how the differentiation between shape and texture information can be fully exploited to create appropriate unit-selection costs and to enhance the video concatenations. The latter is very important since for the synthetic visual speech a careful balancing between signal smoothness and articulation strength is required. Several optimization strategies to enhance the quality of the synthetic visual speech are proposed. By measuring the properties of each model parameter, an effective normalization of the visual speech database is feasible. In addition, the visual joins can be optimized by a parameter-specific concatenation smoothing. To further enhance the naturalness of the synthetic speech, a spectrum-based smoothing approach is introduced.